Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: moved json_dumps call to internal helper #114

Merged
merged 4 commits into from
Jun 5, 2024

Conversation

mj0nez
Copy link
Contributor

@mj0nez mj0nez commented May 15, 2024

As suggested, the call of json.dumps was moved to an internal helper in the StruclogFormatter, which now can be overwritten by inheriting classes.

I was unsure how and if you would like to add a recipe to the docs, but this would be a minimal configuration:

# log configuration
from collections.abc import Callable
from typing import Any

import orjson
import structlog

from ecs_logging import StructlogFormatter as EcsStruclogFormatter
from ecs_logging import _utils


def get_orjson_serializer() -> Callable[[dict[str, Any], bool], bytes]:

    def serializer(data: dict[str, Any], sort: bool) -> bytes:
        return orjson.dumps(
            data,
            option=orjson.OPT_SORT_KEYS if sort else None,
            default=_utils._json_dumps_fallback,
        )

    return serializer


class BinaryEcsFormatter(EcsStruclogFormatter):
    def _json_dumps(self, value: dict[str, Any]) -> bytes:
        # Ensure that the first three fields are '@timestamp',
        # 'log.level', and 'message' per ECS spec
        ordered_fields = {}
        try:
            ordered_fields["@timestamp"] = value.pop("@timestamp")
        except KeyError:
            pass

        # log.level can either be nested or not nested so we have to try both
        try:
            ordered_fields["log.level"] = value["log"].pop("level")
            if not value["log"]:  # Remove the 'log' dictionary if it's now empty
                value.pop("log", None)
        except KeyError:
            try:
                ordered_fields["log.level"] = value.pop("log.level")
            except KeyError:
                pass
        try:
            ordered_fields["message"] = value.pop("message")
        except KeyError:
            pass

        serializer = get_orjson_serializer()

        # Because we want to use 'sorted_keys=True' we manually build
        # the first three keys and then build the rest with the serializer
        if ordered_fields:

            ordered_json = serializer(ordered_fields, sort=False)
            if value:
                return ordered_json[:-1] + b"," + serializer(value, True)[1:]
            else:
                return ordered_json
        # If there are no fields with ordering requirements we
        # pass everything into the serializer
        else:
            return serializer(value, sort=True)


structlog.configure(
    processors=[
        structlog.processors.TimeStamper(key="@timestamp", fmt="iso", utc=True),
        BinaryEcsFormatter(),
    ],
    logger_factory=structlog.BytesLoggerFactory(),
)
# your script
logger = structlog.get_logger()

if __name__ == "__main__":
    logger.info("Hi there!")

Should produce a correct output like:

{"@timestamp":"2024-05-15T21:03:08.796637Z","log.level":"info","message":"Hi there!","ecs":{"version":"1.6.0"}}

Resolves #112

Copy link

cla-checker-service bot commented May 15, 2024

💚 CLA has been signed

Copy link
Member

@xrmx xrmx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@xrmx
Copy link
Member

xrmx commented May 17, 2024

run docs-build

@xrmx xrmx requested a review from basepi May 20, 2024 07:06
@xrmx xrmx enabled auto-merge (squash) May 29, 2024 14:32
@mj0nez
Copy link
Contributor Author

mj0nez commented Jun 5, 2024

Hey, is there something I need to do to trigger the docs build?

@xrmx
Copy link
Member

xrmx commented Jun 5, 2024

run docs-build

@xrmx xrmx disabled auto-merge June 5, 2024 15:45
@xrmx xrmx enabled auto-merge (squash) June 5, 2024 15:45
@xrmx xrmx merged commit e6993eb into elastic:main Jun 5, 2024
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Using orjson for serialization
3 participants